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Abstract 

Beginning  with  the  Better  Buying  Power  initiatives,  the  military  components  have  been 
required  to  perform  full  life-cycle  portfolio-level  affordability  analyses  with  respect  to  major 
defense  acquisition  programs  at  key  milestones.  Prior  to  this,  there  was  no  formal 
requirement  to  reconcile  program  acquisition  baselines  with  resource  forecasts  beyond  the 
five  years  of  the  Future  Years  Defense  Program.  This  paper  discusses  the  analytical 
challenges  associated  with  Affordability  Analysis  and  describes  methods  and  tools  that  the 
Institute  for  Defense  Analyses  is  developing  to  address  these  challenges. 

Introduction 

What  Is  Affordability? 

Affordability  means  conducting  a  program  at  a  cost  constrained  by  the  maximum 
resources  the  Department  can  allocate  for  that  capability.  Many  of  our  programs 
flunk  this  basic  test  from  their  inception. 

— Honorable  Ashton  B.  Carter  (2010),  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 

Affordability  policy  is  about  establishing  the  dollar  amount  the  Component  is  willing  to 
spend  on  the  desired  capability  in  the  context  of  all  other  fiscal  demands  over  the 
long  term. 

— Chad  J.  Ohlandt  (2013) 

Over  the  last  two  decades,  the  Department  of  Defense  (DoD)  has  spent  a  historically 
unprecedented  amount  of  money  on  acquisition  programs  that,  in  the  end,  did  not  deliver 
the  warfighting  capability  they  had  been  intended  to  provide.  For  example,  the  Army 
Acquisition  Review  commissioned  by  the  Secretary  of  the  Army  and  published  in  201 1 
estimated  that 


every  year  since  1996,  the  Army  has  spent  more  than  $1B  annually  on 
programs  that  were  ultimately  cancelled.  Since  2004,  [35%-42%]  per  year  of 
Army  [Development,  Testing,  and  Evaluation]  funding  has  been  lost  to 
cancelled  programs.  (Decker  &  Wagner,  201 1 ) 

The  Army  is  scarcely  unique  in  this  regard.  All  military  components  have  seen 
significant  program  cancellations  with  little  or  nothing  to  show  for  the  billions  spent.  In 
addition  to  cancellations,  many  other  programs  have  been  severely  truncated  or 
restructured,  providing  far  less  national  security  capability  than  had  been  envisioned. 

While  some  programs  have  been  canceled  for  technical  reasons,  or  because  the 
threat  they  were  intended  to  counter  went  away,  others  have  been  canceled  or  curtailed 
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simply  because  there  was  not  enough  money  available  to  pay  for  everything.  Affordability 
simply  means  having  enough  resources  to  be  able  to  finish  the  programs  you  start. 

What  Is  Affordability  Analysis? 

At  the  Milestone  A  Review  [the  DoD  Component  will]  present  an  affordability  analysis 
and  proposed  affordability  goals  based  on  the  resources  that  are  projected  to  be 
available  to  the  DoD  Component  in  the  portfolio(s)  or  mission  area(s)  associated  with 
the  program  under  consideration.  The  analysis  will  be  supported  by  a  quantitative 
assessment  of  all  of  the  programs  in  the  prospective  program’s  portfolio  or  mission 
area  that  demonstrates  the  ability  of  the  Component’s  estimated  budgets  to  fund  the 
new  program  over  its  planned  life  cycle. 

— DoD  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 

System  (USD[AT&L],  2015) 

Affordability  analysis  is  a  DoD  Component  leadership  responsibility  that  should 
involve  the  Component’s  programming,  resource  planning,  requirements, 
intelligence,  and  acquisition  communities.  The  Department  has  a  long  history  of 
starting  programs  that  proved  to  be  unaffordable.  The  result  of  this  practice  has  been 
costly  program  cancelations  and  dramatic  reductions  in  inventory  objectives.  Thus, 
the  purpose  of  Affordability  Analysis  is  to  avoid  starting  or  continuing  programs  that 
cannot  be  produced  and  supported  within  reasonable  expectations  for  future 
budgets. 

—DoDI  5000.02  (USD[AT&Lj,  2015,  Enclosure  8) 

Beginning  with  the  Better  Buying  Power  (BBP)  memorandum  and  directive  of  2010, 
and  continuing  through  BBP  2.0,  BBP  3.0,  and  the  newly  revised  DoDI  5000.02,  Operation 
of  the  Defense  Acquisition  System,  the  Office  of  the  Secretary  of  Defense  (OSD)  has 
defined  and  implemented  a  series  of  new  requirements  for  the  military  components  to 
address  affordability  issues  at  the  inception  of  new  programs  and  at  every  subsequent 
milestone  (DoD,  n.d.)1.  The  required  Affordability  Analysis  should  be  performed  at  the 
portfolio  level,  over  a  planning  horizon  of  decades.  The  guidance  is  explicit:  the 
responsibility  for  Affordability  Analysis  lies  with  the  component  leadership,  not  with  the 
program.  Only  the  component  leadership  has  the  necessary  understanding  of  component 
priorities,  risk  tolerance,  and  resource  forecasts  to  support  the  required  analysis,  and  only 
the  component  leadership  has  the  authority  to  stretch,  truncate,  or  cancel  some  programs  in 
order  to  make  room  for  others. 

What  Tools  Do  Affordability  Analysts  Need? 

Predictive  Costing — “What  If?” 

The  first  fundamental  requirement  for  Affordability  Analysis  is  the  ability  to  consider 
hypothetical  alternative  futures  in  which  the  current  plans  cannot  be  executed.  Current  law 
requires  that  planned  acquisition  costs  must  fit  within  forecast  budgets  through  the  five-year 


1  Better  Buying  Power  (BBP)  1 .0  was  introduced  in  2010  as  part  of  the  DoD’s  Efficiency  Initiative.  The 
stated  objective  was  to  deliver  warfighting  capabilities  needed  within  the  constraints  of  a  declining 
defense  budget  by  achieving  better  buying  power  for  the  warfighters  and  taxpayer. 
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Future  Years  Defense  Program  (FYDP),  but  not  in  the  “outyears”  after  that.  This  leads  to  the 
phenomenon  known  as  “the  bow  wave,”  in  which  the  total  planned  costs  of  all  acquisition 
programs  exceed  reasonable  budget  projections  in  the  years  immediately  beyond  the 
FYDP.  Each  new  President’s  Budget  imposes  fiscal  reality  on  one  additional  year,  pushing 
the  excess  off  into  the  future.  The  new  Affordability  Analysis  requirement  extends  the 
constraints  of  the  FYDP  out  to  a  30-year  planning  horizon,  eliminating  the  bow  wave.  The 
top  line  budget  assumptions  for  each  component  are  provided  by  OSD. 

To  meet  this  requirement,  component  programmers  need  to  be  able  to  understand 
how  annual  program  costs  would  change  under  hypothetical  alternative  schedules.  As  we 
shall  see  below,  this  is  not  an  easy  task. 

Consistency  Checking 

Another  important  requirement  for  Affordability  Analysis  is  the  ability  to  assess  the 
consistency  between  various  affordability  submissions.  The  new  guidance  requires 
Affordability  Analyses  to  support  milestone  decisions  for  specific  programs.  These  generally 
provide  considerable  detail  about  the  program  in  question,  some  detail  about  other 
programs  in  the  same  acquisition  portfolio,  and  minimal  detail  about  other  portfolios. 
Decision-makers  need  to  be  able  to  recognize  whether  new  Affordability  Analyses  are 
consistent  with  past  submissions  by  other  programs  and  portfolios.  They  need  to  be  able  to 
distinguish  updated  information  (e.g.,  new  cost  estimates  or  quantity  requirements)  from 
inconsistent  assumptions.  They  also  need  to  be  able  to  understand  the  implications  of 
reconciling  inconsistent  submissions  using  the  most  authoritative  available  data  for  all 
programs. 

Risk  Modeling 

Because  all  Affordability  Analyses  are  uncertain,  it  is  essential  for  decision-makers 
(both  within  the  components  and  in  the  oversight  community)  to  understand  the  potential 
implications  of  that  uncertainty.  One  important  tool  in  this  regard  is  sensitivity  analysis — the 
ability  to  see  the  overall  consequences  of  changing  specific  assumptions  or  forecasts.  For 
affordability,  it  will  be  important  to  understand  sensitivity  to  cost  estimating  assumptions, 
sensitivity  to  budget  forecasts,  and  sensitivity  to  future  needs  for  not-yet-defined  systems. 

More  generally,  “affordability”  is  not  a  yes/no  question.  It  makes  much  more  sense  to 
think  of  the  probability  that  a  given  portfolio  of  programs  can  be  acquired  within  a  given 
budget,  even  though  both  program  cost  estimates  and  budget  forecasts  are  subject  to 
unknown  errors.  An  ideal  tool  for  Affordability  Analysis  will  support  probabilistic  modeling  of 
individual  program  outcomes,  portfolio  contents,  and  budget  forecasts. 

Visualization 

Finally,  it  is  important  for  any  Affordability  Analysis  support  tool  to  provide  outputs 
that  are  informative  and  understandable.  Numerical  reporting  is  important,  but  data 
visualization  can  greatly  enhance  the  utility  of  any  analytical  tool.  In  particular,  dynamic 
visualizations  that  highlight  the  patterns  of  change  that  occur  as  budgets  grow  tighter  (or 
portfolio  contents  grow)  would  be  very  useful.  To  support  this  requirement,  it  will  be 
necessary  for  the  tool  to  be  able  to  perform  its  “What  if?”  computations  in  near-real  time. 

What  Are  the  Analytical  Challenges  of  Affordability  Analysis? 

Why  is  Affordability  Analysis  hard?  Setting  aside  the  political  challenges  of 
competing  interests  within  a  given  military  component,  it  might  not  be  obvious  that  there  are 
also  substantive  technical  challenges  in  figuring  out  whether  or  not  a  set  of  proposed 
programs  is  affordable.  In  this  section,  we  discuss  three  necessary  enablers  for  effective 
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Affordability  Analysis  and  the  technical  challenges  associated  with  each.  All  three  of  these 
challenges  arise  directly  from  the  nature  of  program  cost  estimates. 

Any  Affordability  Analysis  must  begin  with  cost  estimates  for  the  various  programs  in 
the  relevant  portfolio.  These  estimates  generally  take  the  form  of  annual  quantity  and  cost 
forecasts  that  reflect  the  program  offices’  planned  (or  proposed)  development  and  fielding 
schedules.  If  the  sum  of  these  forecasts  fits  within  the  predicted  available  budget  for  the 
portfolio,  then  the  portfolio  would  seem  to  be  affordable.  But  what  if  it  does  not  fit?  And  what 
about  the  possibility  that  the  budget  forecast  or  the  cost  estimate  (or  both)  might  turn  out  to 
have  been  overly  optimistic?  To  address  those  concerns,  we  need  to  be  able  to  predict  the 
annual  costs  for  production  schedules  other  than  the  planned  schedule — and  that  turns  out 
to  be  rather  tricky. 

Estimating  the  Annual  Costs  of  Hypothetical  Schedules 

The  problem  is  easy  to  state:  “The  program  says  that  the  planned  schedule  of  lot 
quantities  would  result  in  these  estimates  of  annual  lot  costs.  How  much  would  each  lot  cost 
if  we  bought  them  according  to  this  alternative  schedule  of  procurement  quantities  instead?” 
The  DoD  has  detailed  records  of  cost  and  schedule  for  hundreds  of  acquisition  programs, 
going  back  decades.  Given  that  wealth  of  data,  why  is  it  hard  to  figure  out  how  schedule 
affects  cost?  There  are  several  important  analytical  obstacles. 

Obstacle  #1 :  Cause  Versus  Effect 

Consider  three  procurement  programs:  A,  B,  and  C.  Program  A  is  doing  fine,  but  due 
to  overall  budget  reductions,  its  production  schedule  is  going  to  be  stretched,  which  will 
increase  unit  costs.  Program  B  has  just  announced  significant  cost  growth — not  caused  by  a 
schedule  change — that  has  made  its  planned  production  schedule  too  expensive,  given 
other  priorities  in  the  portfolio.  Program  C  has  been  experiencing  integration  issues — its 
electronics  are  going  to  require  a  new  design  that  uses  a  more  expensive  subcomponent, 
which  will  have  to  be  retrofitted  into  existing  units.  This  means  both  a  cost  increase  (due  to 
the  new  component)  and  a  schedule  slip  (to  accommodate  the  new  design  and  the  rework). 

For  all  of  these  programs,  unit  cost  went  up  and  average  production  rate  went  down. 
Causally,  though,  we  have  three  distinct  cases: 

•  For  Program  A,  schedule  stretch  caused  cost  growth. 

•  For  Program  B,  cost  growth  caused  schedule  stretch. 

•  For  Program  C,  technical  issues  caused  both  cost  and  schedule  growth. 

Although  we  are  trying  to  understand  only  the  first  of  those  mechanisms,  we  cannot 
tell  just  by  looking  at  historical  numbers  which  case  was  in  effect — or  whether  it  was  some 
mix  of  all  of  them — for  a  given  program.  We  need  a  way  to  isolate  the  Program  A  effect  from 
the  others. 

Obstacle  #2:  Not  All  Costs  React  the  Same  Way  to  Schedule  Changes 

Since  2006,  Selected  Acquisition  Reports  have  broken  out  cost  forecasts  into 
subcategories:  end-item  recurring  flyaway  costs,  non-end-item  recurring  flyaway  costs, 
nonrecurring  costs,  and  two  categories  of  support  costs.  This  is  very  helpful,  because  we  do 
not  expect  all  of  those  costs  to  react  identically  to  a  change  in  production  schedule.  End- 
item  recurring  costs  should  be  most  directly  affected,  while  nonrecurring  costs  and  non¬ 
spares  support  might  not  be  affected  at  all.  Any  econometric  model  of  how  schedule  affects 
cost  should  take  advantage  of  these  different  cost  categories  and  treat  them  separately 
when  they  are  known. 
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Obstacle  #3:  Limited  Relevant  Data 


Even  if  we  have  multiple  historical  cost  estimates  for  Program  A,  those  past 
estimates  might  not  tell  us  anything  about  the  relationship  today  between  Program  A’s 
production  schedule  and  lot  costs.  Since  those  past  estimates  were  developed,  Program  A 
may  have  changed  in  any  number  of  ways — new  designs,  revised  cost  estimates, 
requirements  changes,  technology  insertions,  planned  product  improvements,  new 
contracts,  new  demands  from  the  field,  and  so  forth. 

Unless  we  could  somehow  correct  for  all  of  the  program  changes  other  than 
schedule,  those  past  forecasts  do  not  tell  us  what  the  estimated  cost  would  be  today  for  that 
previous  planned  schedule.  More  generally,  we  seldom  get  to  see  multiple  proposed 
schedules  (and  their  associated  costs)  for  the  same  exact  program.  Since  we  are  trying  to 
figure  out  how  cost  varies  as  schedule  varies,  this  is  a  major  limitation. 

Obstacle  #4:  Rate  Effects  and  Learning  Curves 

The  cost  of  producing  one  unit  of  a  product  depends  on  the  production  rate.  For  one 
thing,  at  lower  production  rates,  the  indirect  costs  (overhead)  of  the  producer  and  the  fixed 
costs  of  production  get  amortized  over  fewer  units,  so  that  each  unit  bears  a  higher 
proportion  of  those  costs.  In  addition,  there  can  be  logistical  issues  related  to  supply  chains 
and  staffing  that  make  it  inefficient  to  produce  at  low  rates.  We  will  discuss  the  mechanisms 
of  rate  effects  below;  for  the  moment  it  suffices  to  note  that  they  exist.  As  a  result,  the  cost  of 
a  unit  depends  on  how  many  other  units  you  make  in  the  same  year. 

In  addition,  even  at  constant  production  rates,  unit  costs  are  generally  not  constant 
over  the  production  life  of  a  program.  In  particular,  most  procurement  programs  exhibit 
learning  curves,  in  which  the  marginal  cost  of  successive  units  decreases  as  a  function  of 
cumulative  production.  As  a  result,  the  cost  of  a  given  unit  depends  not  only  on  how  many 
other  units  you  make  in  that  year,  but  which  units  (cumulative)  they  are.  In  the  standard 
model,  the  cost  of  the  nth  unit  produced  is  predicted  to  be  Tyri? ,  where  7\  is  a  notional  first 
unit  cost  and  /?  is  a  parameter  that  describes  how  quickly  unit  costs  decline  with  cumulative 
production  (Lee,  1997).  This  formulation  is  often  expressed  in  terms  of  the  learning  “slope,” 

S  =  20,  where  unit  cost  decreases  by  a  factor  of  S  every  time  cumulative  production 
doubles. 

Shifting  units  of  planned  production  from  one  year  to  another  thus  changes  both  the 
production  rates  in  those  years  and  the  portion  of  the  learning  curve  that  falls  in  those  years, 
resulting  in  nonlinear  changes  in  annual  costs. 

Mechanisms  and  Models  for  Schedule  Effects 

There  are  several  competing  theories  about  how  and  why  unit  costs  change  when 
schedules  change.  These  theories  are  not  necessarily  mutually  exclusive,  which  makes  it 
even  trickier  to  figure  out  how  to  combine  them  into  a  coherent  model. 

Fixed  Costs  and  Sticky  Costs 

As  noted  above,  most  of  the  indirect  costs  and  some  of  the  direct  costs  of  producing 
a  weapon  system  are  incurred  per  unit  time,  rather  than  per  unit  produced.  For  example,  the 
costs  associated  with  running  the  program  office  do  not  depend  much  on  the  current 
production  rate,  or  on  how  many  units  have  been  produced  so  far.  Similarly,  the  indirect 
costs  associated  with  contractor  overhead  are  only  a  little  bit  sensitive  to  production  rates. 
What’s  more,  overhead  rates  tend  to  be  “sticky” — they  don’t  generally  adjust  instantaneously 
to  changes  in  work  level.  A  useful  model  of  how  cost  depends  on  schedule  should  be  able  to 
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distinguish  rate-insensitive  costs  from  rate-sensitive  costs,  and  estimate  how  sticky  the  fixed 
costs  are. 

Learning  and  Forgetting 

It  is  not  uncommon  for  unit  production  costs  to  follow  a  standard  learning  curve  for 
most  of  the  life  of  a  program,  but  then  start  to  climb  upward  again  toward  the  end  of  the 
program.  To  account  for  this,  C.  Lanier  Benkard  (2000)  suggested  that  producers  improve  in 
efficiency  by  gaining  “experience”  making  units  (learning),  but  that  this  experience 
depreciates  at  a  constant  rate  (forgetting).  Thus,  early  in  production  (when  cumulative 
quantity  is  doubling  frequently),  or  at  high  production  rates  (when  more  experience  is  being 
gained  per  unit  time),  learning  behavior  dominates.  Late  in  the  production  run,  or  at  low 
production  rates,  the  gains  from  learning  are  visibly  offset  by  forgetting. 

We  investigated  this  model,  and  found  that  it  fits  many  historical  programs  quite  well. 
It  can  also  be  improved  by  combining  it  with  a  fixed  cost  model,  so  that  indirect  and  rate- 
insensitive  costs  are  modeled  separately,  while  rate-sensitive  costs  are  modeled  by  a 
combination  of  learning  and  forgetting. 

Regulatory  Lag 

Finally,  William  Rogerson  (1994)  has  proposed  that  the  interaction  between  unit  cost 
reduction  and  production  rate  can  be  understood  by  looking  at  the  incentives  inherent  in 
how  procurement  contracts  are  awarded.  In  general,  a  new  fixed-price  procurement  contract 
is  awarded  for  each  annual  lot,  with  a  negotiated  unit  price  based  on  the  contractor’s 
demonstrated  historical  costs.  A  contractor  who  invests  in  management  or  tooling  changes 
that  reduce  production  costs  will  only  realize  extra  profits  from  that  investment  until  a  new 
price  is  negotiated — typically  two  or  three  years  later. 

At  high  production  rates,  contractors  have  more  incentive  to  invest  in  reducing 
production  costs,  because  they  will  realize  extra  profits  on  many  units  during  the  two-  to 
three-year  “regulatory  lag”  period  before  the  price  is  renegotiated  downward  to  reflect  the 
lower  production  costs.  As  a  result,  more  potential  cost-reducing  investments  will  be  cost- 
effective  for  them,  given  the  need  to  make  back  the  initial  investment  costs  through  higher 
profits. 

Conversely,  at  lower  production  rates,  contractors  have  less  incentive  to  reduce 
costs,  as  well  as  fewer  available  cost-reduction  alternatives  that  would  provide  the 
necessary  return  on  investment.  If  this  theory  is  correct,  we  should  expect  to  see  less 
learning  at  low  production  rates,  and  more  learning  at  higher  production  rates.  This  is  very 
different  from  the  standard  learning  model,  which  assumes  that  the  learning  curve  slope  is 
an  intrinsic  characteristic  of  the  system  being  produced  and  does  not  vary  with  rate. 

It  is  possible  to  formulate  an  econometric  model  based  on  Rogerson’s  (1994) 
framework,  assuming  a  profit-maximizing  contractor  with  a  notional  pool  of  cost-reducing 
investment  options.  We  can  also  combine  this  model  with  a  fixed-cost  model,  as  we  did  with 
the  learning-and-forgetting  model.  In  theory,  we  could  add  forgetting  to  this  model  as  well, 
but  there  are  serious  limits  to  how  many  model  parameters  can  be  estimated  from  the 
available  data. 

Estimating  the  Impact  on  Schedules  of  Hypothetical  Budgets 

In  order  to  understand  the  impact  of  portfolio  budgets  on  cost  and  schedule,  it  is  not 
enough  to  be  able  to  predict  the  annual  costs  associated  with  a  given  sequence  of  lot 
quantities.  We  also  need  to  be  able  to  predict  how  each  program’s  production  schedule 
might  adjust  to  accommodate  a  budget  that  is  too  tight  for  each  program  to  execute  its 
current  individual  plan. 
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In  a  prior  research  effort  (Weber  et  al.,  2003),  we  used  optimization  to  find  minimum- 
cost  feasible  schedules  within  a  given  budget,  subject  to  constraints  on  minimum  and 
maximum  production  rates  and  latest  permitted  delivery  date.  This  approach  is 
unsatisfactory  for  several  reasons.  First,  the  nonlinear  mixed-integer  optimization  problem  is 
quite  difficult,  requiring  industrial-grade  optimization  tools  and  considerable  time.  Second, 
the  optimized  schedules  are  generally  implausible — they  rely  on  the  portfolio  manager’s 
ability  to  make  trades  among  future  years  in  a  manner  that  is  often  politically  impossible. 
They  also  tend  to  oscillate  between  minimum  and  maximum  rates  for  a  given  program 
unless  additional  constraints  or  penalties  are  imposed.  Finally,  they  give  a  very  misleading 
view  of  the  consequences  of  a  change  in  forecast  budget,  since  the  bulk  of  the  cost 
difference  between  the  current  planned  schedule  and  the  revised  schedule  is  due  to 
optimization  of  the  current  plan,  not  due  to  the  change  in  budget. 

For  the  current  effort,  we  have  chosen  to  use  a  greedy  constructive  heuristic  to 
model  the  potential  impact  of  a  budget  change  on  production  plans  within  a  portfolio.  The 
heuristic  attempts  to  maintain  the  same  proportionate  cumulative  funding  by  program  as  in 
the  plan,  but  is  constrained  by  both  the  budget  and  the  minimum  (and  maximum)  feasible 
production  rates  of  the  various  programs.  The  combination  of  fixed  costs  and  minimum 
sustaining  production  rates  implies  that  it  may  not  be  possible  to  match  the  planned  relative 
funding  levels  very  closely.  This  is  especially  true  if  some  programs  are  near  their  minimum 
sustaining  rates  in  the  original  plan.  Those  programs  cannot  be  stretched  further,  so  that  the 
cost  burden  of  stretching  to  fit  under  the  budget  will  be  borne  disproportionately  by  the  other 
programs.  Optionally,  the  heuristic  can  also  incorporate  user-specified  program  priorities. 

At  present,  the  heuristic  simply  reports  infeasibility  if  there  is  insufficient  budget  to 
produce  each  program  at  its  minimum  sustaining  rate.  Future  versions  may  suggest  quantity 
reductions  where  needed,  based  on  user-provided  priorities  among  programs  and  minimum 
useful  quantities  to  field. 

Affordability  Risk  Analysis 

All  of  the  analytical  challenges  discussed  thus  far  make  the  implicit  assumption  that 
the  estimated  costs  for  each  program  in  the  portfolio  are  accurate.  History  teaches  that  this 
is  not  generally  true;  weapon  system  acquisition  is  notorious  for  cost  growth.  Without  getting 
into  the  thorny  question  of  what  causes  cost  growth,  we  can  nevertheless  ask  what 
implications  cost  growth  has  for  Affordability  Analysis. 

If  we  knew  both  program  costs  and  future  budgets  with  certainty,  affordability  would 
be  a  yes/no  question.  Budget  uncertainty  can  be  handled  to  some  extent  through  sensitivity 
analysis — how  much  do  costs  change  as  a  function  of  future  budget  levels,  and  at  what 
point  does  the  portfolio  become  infeasible  unless  we  reduce  quantities  or  cancel  programs? 
There  is  also  a  time  dependency — one  bottleneck  year  of  low  budget  can  render  a  set  of 
programs  infeasible,  even  if  the  overall  funding  level  over  the  planning  horizon  is  generally 
high. 

Cost  uncertainty  is  even  trickier,  because  we  need  to  know  not  only  how  the  costs 
might  change,  but  why  they  might  change.  Is  the  problem  higher  labor  rates  than  expected? 
Higher  fixed  costs?  Less  learning?  Faster  forgetting?  Some  combination  of  all  of  those?  The 
answers  to  those  questions  will  affect  how  much  further  cost  growth  will  be  incurred  when 
we  realign  production  schedules  to  fit  under  the  budget. 

Because  of  the  complexity  of  how  cost  depends  on  various  econometric  parameters 
and  on  schedule,  there  is  little  prospect  of  an  analytical  model  of  affordability  risk.  Even  if  we 
were  confident  that  we  knew  the  probability  distributions  for  future  budgets  and  for 
procurement  cost  growth  in  each  program,  that  is  not  enough  information  to  allow  us  to 
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derive  the  conditional  distribution  of  resulting  costs  when  all  programs  are  modified  to  fit 
within  a  given  budget.  If  we  also  want  to  be  able  to  model  uncertainty  due  to  new  programs 
arriving  into  the  portfolio,  the  problem  is  even  worse. 

Monte  Carlo  methods  seem  like  the  most  promising  approach  here.  If  we  can 
characterize  the  cost  risk  in  terms  of  probability  distributions  on  the  parameters  of  our 
econometric  model,  we  can  then  simulate  the  future  repeatedly  to  estimate  the  probability  of 
affordability,  the  expected  unit  cost,  the  probability  of  a  Nunn-McCurdy  breach,  the  expected 
time  to  deliver  the  Nth  unit,  and  other  measures  of  interest.  We  can  do  this  for  a  fixed 
budget,  or  we  can  allow  the  budget  to  vary  randomly  at  the  same  time,  including  making 
room  for  future  programs.  The  trick,  of  course,  is  deriving  plausible  probability  models  for  the 
econometric  parameters  and  budgets,  based  on  historical  data. 

APASS — The  Acquisition  Portfolio  Affordability  Support  System 

We  are  currently  developing  a  software  environment  for  Affordability  Analysis  for  our 
sponsors  in  the  Office  of  the  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(OSD[AT&Lj)  Acquisition  Resources  &  Analysis,  which  we  call  the  Acquisition  Portfolio 
Affordability  Support  System,  or  APASS.  The  purpose  of  APASS  is  to  provide  an 
environment  for  evaluating  and  tracking  the  affordability  of  portfolios  over  time.  APASS 
features  are  focused  on  five  specific  areas  of  support:  tracking,  forecasting,  risk  analysis, 
reporting,  and  visualization. 

Tracking 

APASS  will  provide  both  a  “living”  operational  view  and  a  historical  record  of  portfolio 
affordability  and  cost  and  schedule  data.  This  combination  will  support  both  current 
decision-making  and  assessment  of  historical  trends  and  changes. 

Affordability  analyses  are  generally  performed  on  behalf  of  individual  programs  for 
use  at  milestone  reviews.  These  analyses  generally  feature  detailed  cost  and  schedule 
information  for  the  program  under  review,  aggregated  cost  and  schedule  information  for 
other  programs  in  that  portfolio,  and  at  best  top-line  summary  data  for  other  portfolios. 
APASS  will  allow  managers  to  remember  the  information  submitted  at  prior  reviews, 
compare  that  information  against  the  new  information,  and  identify  any  inconsistencies  or 
changes.  In  particular,  APASS  will  support  comparison  of  data  sources  with  slightly  different 
portfolio  definitions.  APASS  will  also  be  able  to  compare  new  submissions  against  routine 
DoD-wide  data  submissions,  such  as  the  annual  Selected  Acquisition  Reports  (SARs)  or 
President’s  Budget  justification  exhibits. 

APASS  will  also  allow  analysts  to  merge  multiple  submissions  into  user-defined  data 
sets.  Authoritative  data  from  multiple  sources  could  thus  be  combined  to  create  a  “single 
integrated  affordability  picture”  across  all  programs  and  portfolios.  This  would  let  managers 
maintain  a  living  representation  of  the  current  best  forecast  of  the  acquisition  future,  with 
which  they  could  explore  the  implications  on  that  future  of  alternative  budgets,  new 
programs,  or  cost  growth. 

Forecasting 

The  primary  analytical  function  of  APASS  is  to  predict  the  annual  quantities  and 
costs  for  all  programs  in  a  portfolio,  conditioned  on  a  portfolio  budget  other  than  the  current 
planned  budget.  This  function  relies  heavily  on  both  the  econometric  modeling  and  the 
schedule  adjustment  heuristic  described  above. 

Figure  1  shows  an  APASS  screenshot  of  a  notional  current  production  plan  for  a  set 
of  portfolios.  FYDP  years  are  shown  in  darker  hues;  outyears  are  lighter  shades  of  the  same 
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color.  If  we  drill  down  into  a  single  portfolio,  we  can  see  the  individual  program  plans  and  the 
projected  portfolio  budget  (Figure  2).  The  combined  plans  exceed  the  projected  budget,  so 
we  invoke  the  “fit  to  budget”  heuristic  to  extrapolate  how  the  portfolio  might  adapt  to  the 
lower  budget.  The  result  is  shown  in  Figure  3.  In  this  example,  the  total  procurement  cost  for 
the  portfolio  increases  significantly.  The  percent  increase  in  unit  cost  is  not  shared  evenly 
across  programs;  those  programs  that  finish  quickly  or  that  are  already  near  their  minimum 
production  rates  will  not  be  as  strongly  affected  as  those  that  have  more  flexibility  to  stretch. 


Figure  1.  Army  Procurement  Portfolios 


Figure  2.  Portfolio  With  Budget 
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Figure  3.  Portfolio  Adjusted  to  Fit  Within  Budget 


This  is  the  fundamental  analytical  tool  in  APASS;  all  other  analyses  rely  on  this  ability 
to  extrapolate  costs  and  schedules  in  response  to  different  levels  of  available  funding.  The 
cost  forecasts  produced  by  the  econometric  models  and  extrapolation  heuristic  are  not 
“budget  quality”  cost  estimates,  but  they  provide  credible  extrapolations  of  the  type  and 
magnitude  of  cost  and  schedule  impact  that  would  be  implied  by  alternative  scenarios. 

Risk  Analysis 

The  initial  implementation  of  APASS  will  focus  on  sensitivity  analysis,  particularly 
with  respect  to  changes  in  budget  or  portfolio  contents.  It  is  particularly  important  for 
decision-makers  to  understand  how  program  costs  and  delivery  schedules  might  change 
under  various  budget  scenarios,  or  in  the  event  that  a  new  high-priority  program  is  added  to 
a  portfolio.  It  will  also  be  possible  to  use  APASS  to  assess  the  sensitivity  of  all  programs  to 
cost  growth  in  a  particular  program. 

Future  versions  of  APASS  will  include  Monte  Carlo  modeling  capabilities,  to  simulate 
the  potential  consequences  of  multiple  simultaneous  sources  of  uncertainty.  The  output  from 
these  simulations  would  include  probability  distributions  on  unit  cost  and  delivery  timelines 
for  individual  programs,  as  well  as  correlations  among  programs  within  a  portfolio. 


Reporting 


The  analytical  capabilities  of  APASS  will  only  be  useful  if  they  are  able  to  provide 
users  with  the  information  they  need  to  support  acquisition  decisions  and  oversight.  To  that 
end,  APASS  will  include  both  standard  and  customizable  reporting  of  both  cost  and 
schedule  baselines  and  derived  affordability  data. 

Because  APASS  incorporates  data  from  multiple  sources,  including  speculative 
analyses,  its  reporting  requirements  will  be  somewhat  more  varied  than  those  of  typical 
acquisition  data  repositories.  In  particular,  APASS  will  need  to  be  able  to  merge  conflicting 
data  sources  to  produce  “best  guess”  forecasts  that  draw  individual  program  forecasts  (and 
their  corresponding  econometric  parametrizations)  from  separate  sources,  using  the  current 
most  authoritative  source  for  each.  This  merged  forecast  will  typically  differ  from  each  of  its 
source  submissions  in  some  ways.  It  will  therefore  be  particularly  important  for  APASS  to 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-92- 


produce  reports  that  highlight  the  differences  between  forecasts  of  the  same  program(s) 
from  different  sources.  These  comparison  reports  will  also  be  useful  for  describing  the  unit 
cost  and  schedule  impacts  of  various  possible  future  portfolio-level  budgets,  as  extrapolated 
by  the  schedule  adjustment  heuristic. 

Visualization 

As  a  complement  to  numerical  reporting,  APASS  will  also  provide  graphical 
visualizations  of  various  baseline  data  and  comparisons.  To  begin  with,  APASS  will  be  able 
to  produce  the  various  graphical  formats  for  presenting  Affordability  Analysis  information 
that  are  recommended  in  the  Defense  Acquisition  Guidebook,  as  revised  in  June  2013. 
APASS  will  also  provide  the  ability  to  view  animations  of  the  sensitivity  of  a  given  portfolio  to 
program  cost  growth,  changes  in  budget  level,  or  introduction  of  additional  programs. 
Coupled  animations,  showing  both  the  overall  “sand  chart”  of  program  spending  over  time 
and  an  aligned  display  of  the  absolute  or  relative  change  from  the  baseline  by  program,  will 
allow  analysts  to  grasp  the  implications  of  alternative  scenarios  much  more  quickly  than 
when  using  individual  graphics  or  numerical  displays. 

Summary  and  Conclusions 

Program  life-cycle  affordability  is  a  cornerstone  of  DoD  acquisition  planning. 
Ultimately,  the  goal  of  Affordability  Analysis  is  to  supply  decision-makers  with  the  best 
available  information  and  analysis  about  defense  acquisition  programs  so  that  they  can 

•  allocate  defense  resources  efficiently  and  effectively,  both  within  and  across 
programs; 

•  consider  the  full  range  of  cost  and  schedule  alternatives; 

•  understand  and  manage  acquisition  risk  at  the  program,  portfolio,  and 
Service  level;  and 

•  avoid  the  time  and  money  wasted  by  starting  programs  that  cannot  be 
completed. 

Affordability  is  the  measure  that  allows  decision-makers  to  allocate  appropriate 
resources  to  their  future  operational  requirements.  Affordability  is  not  a  “yes  or  no”  attribute; 
it  is  the  degree  to  which  uncertain  future  resources  can  be  expected  to  permit  execution  of 
current  and  future  programs.  This  degree  is  best  characterized  in  terms  of  probabilities  and 
sensitivity  to  deviations  from  current  plans. 

The  Better  Buying  Power  (BBP)  memoranda  and  the  newly-revised  DoDI  5000.02, 
Operation  of  the  Defense  Acquisition  System,  from  the  OSD  have  defined  and  implemented 
new  requirements  for  the  military  components  to  address  affordability  issues  at  the  inception 
of  new  programs  and  at  every  subsequent  milestone.  This  Affordability  Analysis  is  to  be 
performed  at  the  portfolio  level,  over  a  planning  horizon  of  decades.  The  responsibility  is 
given  to  component  leadership. 

In  order  to  plan  their  acquisition  strategies,  the  components  need  to  be  able  to 
assess  affordability  over  a  wide  range  of  future  scenarios  without  needing  to  return  to  the 
program  offices  or  the  component  cost  estimators  for  new  cost  estimates  every  time  a 
different  portfolio  plan  is  considered.  At  the  same  time,  OSD  oversight  organizations  need  to 
be  able  to  reconcile  current  affordability  analyses  with  past  analyses,  assess  the  sensitivity 
of  current  forecasts  to  potential  disruptions,  and  characterize  the  level  of  risk  inherent  in 
current  plans,  in  order  to  fulfill  their  oversight  responsibilities  with  respect  to  acquisition. 

APASS  is  being  implemented  as  a  tool  to  support  both  Affordability  Analysis  as 
required  by  current  DoD  regulations  and  OSD  oversight  of  affordability  planning  across  the 
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military  components.  To  these  ends,  APASS  will  combine  econometric  modeling  of 
procurement  costs  as  a  function  of  schedule,  extrapolation  of  the  effects  of  alternative 
budgets  on  portfolios  of  programs,  and  fusion  of  data  from  multiple  divergent  sources. 
APASS  will  also  provide  the  reporting  and  visualizations  services  necessary  to  make  these 
analytical  capabilities  practically  useful  on  a  day-to-day  basis. 
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